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EVALUATION OF E-LEARNING 


Many of UNC’s E-Learning efforts are new and an evaluation of their effects is premature. Despite this, 
evaluating learning outcomes in online education may provide insight for future decisions. 

In order to evaluate these courses, UNC General Administration’s Policy & Funding Analysis Division 
used a statistical technique that allows one to analyze the relationships between different sets of 
variables and outcomes of interest. To create a meaningful comparison group, the sample for our 
analysis only examined courses offered both online and face-to-face. Below is a brief outline of the 
process followed by two images of our key findings. For further information about the study, see 
Appendix A: Quantitative Technical Notes. 

Process for the Evaluation 

1. Create a sample from 2011-12 academic year using undergraduate course sections institutions 
offered both online and face-to-face. 1 2 The sample covers approximately 50% of all online sections. 

2. Conduct separate analyses for the outcomes of “D or F” and “Withdrawal or Incomplete” to 

2 

determine the influence of online delivery across different fields and class level. 

3. Calculate the predicted probabilities of the outcomes on data that simulates all possible combination 
of variables of interest, holding others at the mean. 


How to Interpret the Graphs 


The figures are the result of the regression analysis that allows the “effect” of online to vary 
across class level (Freshmen) and Field. To present the interactions among these variables, 
predictions were generated over simulated data that account for the other independent variables in 
the model. 

• The Black Shades represent Face-to-Face predictions. 

• The Gold Shades represent Online Courses. 

• Circles represent Freshmen and triangles Upper-classmen. 

• The circles and triangles represent the estimates from the predictions. 

• The lines are the corresponding uncertainty (95% confidence intervals) surrounding the 
predictions. 

o When the confidence intervals overlap, we cannot statistically determine the 
difference of online across modes of delivery and/or class level. 


1 These pairs may be taught my multiple instructors and a single student may be in the sample multiple times. 

2 These statistical models control for students’ average GPA in all other classes that academic year, course load, gender, race, 
term, and whether the student received a Pell grant. 
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Figure 1: Interpreting the Data: Predicted Chance of Receiving a D or F, by Mode of Delivery and Selected Fields (Undergraduates, 2011-12) 



How to read analysis: 

Agriculture : No statistical significance between online sections and face-to-face sections. This is evident because the gold lines overlap the black lines, 
circles, and triangles. 

Math & Stats’. Significant difference between online and face-to-face sections, students have a much higher probability of receiving a D or F than do 
students enrolled in face-to-face counterparts. No difference between freshmen and upper classmen within mode of delivery. 

Physical Science’. Face-to-Face delivery has a higher probability of receiving a D or F than online delivery methods. No difference between freshmen 
and upper classmen within mode of delivery. 

Protective Services: Online freshman have a higher probability of receiving a D or F than other students - including both online and face-to-face upper 
classman and face-to-face freshmen. 
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Figure 2: Predicted Chance of Receiving a D or F, by Mode of Delivery and Field (Undergraduates, 2011-12) 
Shaded areas indicate statistically significant differences 



Results of D and F analysis: 


Population sample: Only courses offered both online and face-to-face are included in the analysis. This sample includes approximately 50% of all online 
courses. 

Statistical significance: 14 of the 27 fields show statistically significant differences between online and face-to-face delivery. Of these, the penalty for being a 
freshman online is different for 6 fields. 

Control Variables'. Students’ average GPA in all other classes that academic year, course load, gender, race, term, and whether the student received a Pell 
grant. 
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Figure 3: Predicted Chance of Receiving a Withdrawal or Incomplete, by Mode of Delivery and Field (Undergraduates, 2011-12) 
Shaded Areas indicate statistically significant differences 
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For most disciplines in this analysis, the data show that students are more likely to 
withdraw or fail to complete an online class. Existing research on the effectiveness of online 
instruction is - at best - mixed. However, this analysis with UNC specific data furthers and 
supports previous, studies using smaller samples from within the state . 3 Working with 
campuses, UNC GA would like to explore the many possible factors influencing student success 
in online courses. Including, but not limited to: student preparation for online courses; 
professional development provided to and engaged by faculty; course conduciveness to next 
generation modes of delivery; and the relationship between online and part-time/contingent 
faculty. 


3 Wilson, D. and Allen, D. “Success rates of online versus traditional college students” Research in Higher 
Education Journal 
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APPENDIX A: TECHNICAL NOTES FOR QUANTIATIVE 

METHODOLOGY 


The following appendix describes the approach and technique used to evaluate course success by 
method of delivery and provides tables of descriptive statistics and the results of the two models 
presented in the report. 

To evaluate the success of online courses, we created a sample that matched online courses with 
the same face-to-face courses , matching by section and institution for the 201 1-12 academic 
year. 4 By restricting our analysis to this sample, our evaluation focuses on the difference in 
mode of delivery. 

To support UNC GA’s co mm itment to the redesign of courses with a preponderance of D’s, F’s, 
or Withdrawals (DFW), we used these as our outcomes of interest. Because online education 
D’s/ F’s and Withdrawals would have different policy solutions, we elect to estimate these 
outcomes separately. 

Beyond mode of delivery, we know that outcomes vary across discipline and field, and are 
driven by many student level characteristics. To account for the former, we interacted the 
variable for online with variables for field/discipline and whether or not the student was a 
Freshman. Additionally, we include other variables found in the social science literature (term, 
gender, underrepresented minority 5 , Pell grant) and a variable that is the average of the student’s 
grades in every other course that academic year. The following equation represents our models 
which use a logit link function: 

Pr(Tj = 1) = /? 0 + Piimean graded) + /? 2 ( summer ^ ) + /? 3 ( female ^ ) + /? 4 (race) + 

/ 3 5 (pell granti) + /? 6 ( course loadf) + fi 7 (course load squared f) + /? 8 (online students f) 
p 9 (onlinei) + (3 10 (freshmani) + f^onlinei * freshman i ') + 

/ 3j(fieldij ) + p k (onlinei * field t f) + /?;( freshmani * field if) + 

(3 m (onlinei * freshmani * fieldy ) 

Where Yiis the outcome of interest, p are the covariates in the model, k indexes the field, and the 
last elements represent the interaction terms. 

Because our models use Maximum Likelihood Estimation (MLE) and interactions, we calculate 
the predicted probabilities and their associated 95% confidence intervals through simulating all 
possible combinations of variables in the model. These simulations are run holding all non- 
interacted variables at their sample mean. 


4 The sample uses merged elements from the Student Data File, Financial Aid File, Course Grade File, and 
Course Grade Extension File. 

5 Race is a dummy variable indicating if a student is Black, Pacific Islander, non-U.S. residents, Latino, Native 
American, Two or more races, and unknown race. 
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